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ABSTRACT 

Effectiveness ratings of principals by researchers 
have frequently been hampered by relying on teachers 9 
one-dimensional , research-triggered "global" ratings. This paper 

reports a 1979-80 British Comumbia superintendents * study geared 

toward multidimensional subjective assessments of 266 administrators, 
and concentrates primarily on ways in which superintendents make 
their evaluations. The first, section deals with data--including 
distributions of general evaluator ratings and of ratings by 
positions—and with evaluation strategies . The next section presents 

a random sample of 77 reports and examines the aspects on which 

evaluators focus as they consider principals' schools, rather than 
their behaviors. Professional background, as well as descriptors 
(adjectives) describing the evaluated, are addressed in the third 
section, although it is chiefly given to analyzing the number (290) 
and variety of principals 9 actions in terms of a ^-dimensional 
framework. The next section relates differences between reports of 
differently rated principals, and conclusions are drawn in a closing 
section. (KS) 
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The measurement of effectiveness in principals may be 
said to have been a persistent problem, both In the long 
tradition of research in school administration and in the 
recent spate of "school effectiveness" studies; As Bridges 
(1982) notes, researchers who have attempted to assess the 
effectiveness of administrators have relied heavily on 
ratings made by teachers; Such ratings, he points but, have 
usually been global ratings, and they have in most cases 
been used in relation to a single dimension such as 
•interpersonal effichncy", "instructional supervision" or 
"communication". 

A further concern about effectiveness ratings is that 
they are, for the most part, what light be called "research- 
triggered"" that is to say that they have been given in 
response to a researcher's request to do so, and they have 
not been "real" in the sense that they might have been if 
they were given as part of the respondent's working life. 
This point is echoed by Rowan et si. (1983) who note that the 
typical one-dimension rating of effectiveness is not one 
which corresponds to the mult I -dimensional subjective 
assessments which practitioners make in their working world. 
Rowan et al. go on to suggest that it would be useful to 
develop grounded definitions of "effectiveness" that reflect 
practitioners' subjective understandings of the term. 

This paper reports a study which sought specifically to 



examine one source of the subjective understandings held by 
school superintendents of the effusiveness of 
administrators, The data consisted exclusively of reports 
written by British Columbia school superintendents, in 
conformity with the Act and regulations, on administrators 
In their districts. 

These reports are clearly sensitive documents and it 
was drily subject to impressively stringent conditions of 
confidentiality that I was allowed to use them. The 
agreement was that I would have access for a limited time to 
copies of these reports from which all identification of 
school, school district and the names and personal data of 
both evaluator and subject had been removed. Moreover, I was 
not permitted to make any further copies of the reports and 
was required to return all of them to the Ministry by a 
specified date. 

* 

While these restrictions were to some extent Irksome, 
they were, I think, eminently sensible for a first-time 
project with highly sensitive documentary data, Indeed, 
access to the data at all was an excellent demons trst ion of 
the Ministry's willingness to assist research; The 
conditions did, however, impose some limitations on the 
project. No comparison between districts was possible, no 
means was available of grouping reports according to their 
authors, no analysis of biographical data concerning the 
subjects was possible. Finally, and most importantly, no 
follow Up access to the data was possible once the al iottud 
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time period had passed. 

For my pur P° SflS - however, what i was left with was 
suitable. My primary interest was to find out what it is 
that superintendents focus on when they evaluate their 
subordinated. What do they see™ to mean when they indicate 
that a particular school principal is good, bad or 
indifferent* A Merest was to determine If 

Possible whether different facets of administrative 
performance or style were reported in the case of good and 
less good administrators respectively, 

I very deliberately approached the data with as few 
Prior conceptualizations as possible. The general guiding 
question was -What is in these reports?" The only initial 
Points of specific focus beyond that were those derived from 
a coarse view of evaluation, namely, "How do the evaluators 
collect data?", -how do the evaluators indicate their 
criteria?', "Do the evaluators obtain their subjects' 
agrrement on criteria?', -Do the evaluators say where 
improvements are needed and, if so, how?-, All other 
findings emerged from close scrutiny of the words written by 
the evaluators and from allowing those words to suggest 
other questions which might be asked, 

In what follows the results of the exploration are 
presented in four sections; The first dials with the nature 
of the data and the superintendents' evaluations, This 
sectiSn draws on the complete data base of 266 reports-ail 
the reports" written on administrators In the province In 
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1979 and 1980 ' The next three sections use the data of a 
iMple of the reports (N»77) dealing with school principals 
only and excluding reports en other kinds of administrator. 
The reason for using only a sample of the reports was that 
it became' Increasingly apparent that detailed and 
Painstaking scrutiny of the texts of the reports was 
essehtiai and the limited time for which the documents were 
available made this kind of analysis of ,11 2 66 reports 
impossible. These three sections deal respectively with the 
features of the schools run by ' the principals who were 
evaluated, the features of the principals themselves, and 
the differences between what is reported for differently 
rated principals, A final section presents some conclusions 
and speculations about the results of the exploration. 

THE NATURE OF THE DATA AND THE EVALUATIONS 

Two hundred and sixty-six administrators were evaluated 
m British Columbia In 1979 and i960. Twenty-one of them 
were women; two hundred and one of them were school 
Principals, Since the personal data concerning the subjects 
had been removed from the reports, It was necessary "to 
determine what positions they held from a number of kinds of 
reference In the text, For this reason a small number of 
reports were designated as 'Principal-grade level not 
clear-, A number of central office &m\mm had been 
evaluated and these included one Director and a" Somber of 
coordinators, supervisors and consultants. The composition 
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Table 1 



of the group of subjects Is shown In Table I, 



distriiution op subjects n administrative position and sex 



The reports on these subjects are in a form which has 
three main components! 

(1) A printed section, using the top quarter or so 
of the page and providing the basic biographical 
data about the person being evaluated. This 
includes his or her name', birthdate, length of 
time in present position, qualifications, and so 
on, Oh the reports as J received them this entire 
section had been removed, 

(2) An anecdotal report. These anecdotal reports 
in the present set ranged In length from less than 
brie page to fourteen pages. 

(3) A final summary judgement by the evaluator, 
• This is required on a seven-point scale in which 

("Excellent and 7«Poor, although some districts 
declined to comply with this scale, preferring to 
rate their staff only as "Satisfactory' or "Less 
than Satisfactory". 



POSITION 



DISTRICT STAFF 

PRINCIPAL SI SECONDARY 
PRINCIPAL JR. SECONDARY 
PRINCIPAL JR.-SR. SECONDARY 
PRINCIPAL R-12 
PRINCIPAL K-IO 

PRINCIPAL R-7 
PRINCIPAL PRIMARY 

PRINCIPAL, GRADE LEVEL NOT CLEAR 

VI^lNCtPAL, SECONDARY 
VICE PRINCIPAL, ELEHENTARI 

OTHER 
TOTAL 




266 ton 



Table 2 shows the distribution of the ratings assigned 
by evaluators over the whole set of reports, Three points 
should be noted about this table; First, although the 
tallies shown suggest that each subject received a single 
rating, there were In fact twenty-eight reports which 
concluded with a double rating (e.g. 1-2, 5-6). In all these 
cases, the rating used in the display In Table 2 Is either 
the higher of the two ratings where the lower was 5, or the 
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Table 2 



DISTR18UTI0H OF RATINGS ASSIGNED TO SUBJECTS 
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lower of the too ratings where the higher was S, Thus "5«S" 
would be shown as "3" and "5-6" as "6"; The Second point to 
be kept in mind in interpreting the table Is that the 
category "Satisfactory" (5) is used differently by different 
evaluators. Some use it as one of only two possible ratings 
(the other being "less than satisfactory"), others use It to 
mean that which is suggested by its numerical value (I.e. 
"not as good as the top four categories, but better than the 
bottom two"), The tone of some of these reports could permit 
Inferences to be drawn about what the evaluator regards as 
the "true" quality of the subject. Such inferences were not 
used in constructing fable 2, but to follow them might lead 
one to increase the rating of about haif of these 
"satisfactory" reports to "very satisfactory", "good" or 
even "very good". Filially, it is probably not possible to 
assume a Fine comparability between the ratings assigned. 
Several ratings of "3", for example, come at the end of a 
report whose overall tone suggests that the sabject is 
better than "good" by comparison with what Ib said of others 
of others who score similarly. 

The way subjects are distributed both by position and 
rating Is shown In Table 3. An exhaustive analysis of the 
data in this table Is beyond the scope Of this paper, but it 
might be noted that "District Staff" account for 35X of ail 
the "excel lent" ratings, but also Include one "less than 
satisfactory". Separating out secondary principals from 
those who enrol elementary grades shows that 53% of the 
former are rated "l" or "2", whereas the latter Have 33.515 
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Of their number In these categories, It might be noted also 
that there Is a reasonably •norma! 11 distribution of ratings 
in those positions which Include the largest number of 
subjects. 

it will be recalled that the few points of specific 
focus in the first analysis of the data were those derived 
from asking questions about the nature of the approach to 
evaluation revealed In the reports (c. f. page 3 above). The 
findings may be briefly summarized. The majority of 
evaluators (66%) do not describe their method of data 
col lection and most (81%) appear to use their own 
observations as a data source, rather than asking questions 
of others. Although more than three quarters of the 
evaluators (78%) do not state their criteria for evaluation, 
most of those who do state them appear also to make a point 
of discussing them with their subjects. Some 40% of the 
evaluators say where improvements are heeded, and more than 
three quarters of this group specify also how the 
Improvements should be made, 

Before concluding this overview of the data and the 
nature of the evaluations, it is perhaps worth mentioning 
one feature that emerged, not as a result of asking a 
specific question, but as a result of noting that some 
reports contained descriptions (more or less detailed) of 
the context in which the subject worked; The flavour of 
these descriptions varied and it was quite clear that some 
had been included because the context appeared to provide 
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what might be called mitigating circumstances. About one 
quarter of all the reports contain material of this kind. 
Mitigating circumstances include a variety of factors 
ranging from Acts of God such as fire and flood to temporary 
inadequacies of the physical plant, but the most commonly 
mentioned are those related to the kind of staff which 
administrators find themselves with and the kind of, 
community in which they must work.' 

The evaluators then, may or may not mention mitigating 
circumstances, they may or may not describe their criteria 
and methods of data collection, they may or may not 
recommend how improvements are to be made. What they will 
certainly do, however, if they are evaluating a school 
principal, is to consider not only the principal and his or 
her performance, but also the school itself which is being 
administered. The next section of the paper examines what It 
is that evaluators appear to focus on when they consider the 
administrator's school as distinct from the administrator's 
personality, behaviour or style. 



■A mitigating circumstance for one evaluatCr may be a 
cHa|1enge__to_be met fprjnother. the.anaylsis.requires soma 
fine Judgements, but there seems little doubt about the 

e ^. ht ..9. f . _?y*iy**Pr ! _l* s ff j l very experienced and 
considerable effort is required on the Principal's part to 
encourage them to participate in in-service.' 

13 



9 



THE SCHOOL AND ITS FEATURES 



From the 201 reports on school principals a sample of 
77 reports was randomly selected. In this sample no fewer 
than eighty-seven different aspects of the schools 
administered by the subjects were referred to. Each of these 
references was rioted and then coded In one of seven ways: 

1. Aspect has deteriorated 

2: Aspect needs Improvement 

3. Aspect mentioned neutrally, there is no problem 

4. Aspect anywhere from satisfactory to very good 

5. Aspect is outstanding 

6. Aspect has improved (A coding of 6 d|d not 

t ually stand alone since thestate which the 
improvement had led to was also noted) 

7. Aspect noted by the evaluator as not present. 

Codes 1 and 2 were taken to constitute an unfavourable 
mention. Codes 3, 4 and 5 were regarded as favourable 
ment ions . A code of 6 was regarded as favourable or 
unfavourable dependng bh its state after the improvement 
noted . 

The frequency of mention of any particular aspect may 
be seen as one measure of the extent to which that aspect 
provides a commonly held focus for superintendents across 
the province Table 4 displays those aspects (referred to as 
•Items" in the table) which were most frequently mentioned. 
Only two aspects of the school featured iri more than half of 
the sample reports. "Supervision of Teaching Staff and 
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M Cbrnmunicat ton with Parents*. Three aspects were found to be 
mentioned In forty to fifty per cent of the reports, 
"Statement of School Philosophy", "Communication with 
Community", and "Overall Educational Climate or Tone". The 
remaining eighty-two aspects of the schools were each 
referred to In fewer that 40% of the reports. 

A classification of the eighty-seven Different aspects 
was made which resulted In six categories: 

1. Physical Plant and Its Use 

2. Routines and Procedures 

3. The Programme and Its Organization 

4. The Staff 

5. Community Relations 

6. District Relations 

Category 3 (The Programme and Its Organization) was broadly 
defined to Include extra curricular activities and other 
Items which 'might be said to contribute to the overall 
educational climate of the school. 

dust as 'the frequency of mention of individual aspects 
of the school could be regarded as constituting a measure of 
the extent to which they provided a common focus for 
superintendents, so the number of reports in which the 
elements of each of the six categories were mentioned might 
be seen as a measure of the salience for evaiuators of each 
category. By this measure Th e Programm e and its Organization 
is the most salient of the dimensions, being referred to by 
one or more of its elements in 86% of the reports in the 
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sample. Seventy-eight per cent of the reports refer to 
Routines and Procedures , Staff , and Community Relations , 
Less salient is Physical Plant (4455 of the reports); and 
relatively infrequent is mention of District Relations ( 14% 
of the reports). 

S different measure of the salience of dimensions for 
the evaluators is provided by an examination of the 
frequency with which elements in the categories are 
mentioned, Columns 4 and 5 of Table 5 show the number of 
mentions of items in each category and the rank order of 
categories on this measure* By this measure also, The 
Programme and Its Organization is dominant. In examining the 
ranks of the categories It is perhaps worth remembering 
that, although Staff ranks only fourth here, the item 
"Supervision of Teaching Staff" was the single most 
frequently mentioned item. It will also be recalled that 
items from The Programme arid its Organization and Community 
Relations were the only other ones to be merit ibned in over 
40% of the sample; 

Columns 2 and 3 of the table show the number of 
favourable and unfavourable mentions respectively in each 
category, The last column of the table shows the ratio of 
favourable to unfavourable merit Ions. Thus, 1 for every 
unfavourable mention in the area of Community Relations, 
there are more than twenty-three favourable mentions; 
whereas for every unfavourable mention in Programme and its 
Organization there are drily slightly over eleven favourable 
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ones; The items concerned with Staff clearly call forth a 
greater proportion of unfavourable mentions than those of 
any other category, 

The finding that evaluators of principals consider not 
only the principal but also the school which he or she runs 
Is not at all surprising (indeed, to Have found otherwise 
would have been strange), but what seems to emerge from this 
analysis is that certain aspects of the school receive more 
attention than others; The apparent salience in this regard 
of The Programme and its Organization may be in part c 
function of the broad definition of this category (which 
resulted in rtiore elements of the school being included in it 
than in any other category), but it also reflects the common 
sense notion that what is important to the evaluator is the 
way the principal's school does what it is supposed to do. 
What also emerges from this analysis is that some Rinds of 
things which go on in the school are more likely than others 
to make a favourable impression on the evaluator, Thus the 
ratio of 23.6:1 for favourable:unfavourab1e mentions of 
Items in the area of Community Relations seems to suggest 
that the principal who pays attention to these relations 
stands a good chance of being commended for it. The work he 
or she does with regard to Staff, however, comes in for much 
more critical appraisal. 
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THE PRINCIPAL; ATTRIBUTES AND ACTIONS 

It appears from the reports examined that evaluatbri 
consider the personal attributes of principals, including 
the way they work, as well as what they do. The following 
paragraphs discuss first the way in which the background and 
the person of the principal are described and second, the 
kind of actions which are referred to by the evaluators, 

The Principal: The Person 

Professional background . Several aspects of the 
principal's professional background stand an equal chance of 
being mentioned or not mentioned. Thus, previous experience, 
professional development activities and professional 
activities (such as serving on committees) are mentioned in 
about one half of the reports and pot mentioned in the other 
half; Whether or not any one of them is mentioned seems to 
depend more on the inclination of jthe evaluator than on 
anything else. Three quarters of the reports do mention the 
number of years the subject has been in his or her present 
position and over eighty per cent of the reports do not 
mention the kind of professional development acquired 
through university cbursewbrk--a finding consistent with 
Other work which has suggested that practitioners have often 
viewed formal training as inadequate in the preparation of 
school administrators (Pltner, 1982), 

One particular kind of attribute began to stand out as 
the reports were read in more and more depth, Its salience 

20 
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emerged riot because of the frequency of its Sent ion; but 
because of the way it was mentioned. It is far from being a 
clear-cut attribute, and a far from clear-cut label was 
coined for it. The name chosen was Personal-Professional 
Awareness and Competence . 

For a statement in the reports to be classified as a 
mention of Personal-Professional Awareness and competence, 
it had to be one which was not made solely with reference to 
what the principal did in his or her school. It was a 
mention which either stated or clearly implied that the 
particular area of awareness or competence was a 
professional interest which went beyond the subject's 
working context and formed part of his or her intellectual 
life. Frequently, but not always, it appears to be 
associated with the subject's being In demand as a resource 
person, 

This Personal -Professional Awareness and Competence is 
mentioned In twenty-three of the seventy-seven sample 
reports, and two things in particular are interesting about 
it: first, It seems to be associated in some way with the 
rating of the principal and, second, it seems to consist of 
three separate elements which form a hierarchy, If hot a 
scale. The three elements are shown in Table 6, which also 
shows the number of times each element is mentioned in the 
reports. The hierarchical flavour of the elements is given 
by the movement from awareness and concern about philosophy, 
values and one's purpose in life, through an interest and 



fable 6 



TYPES Of PERSONAL-PROFESSIONAL AWARENESS AND CONPETENCE 



A HAVING A PHILOSOPHY, VALUES, SENSE OF PURPOSE IN LIFE 



B HAVING PERSONAL INTEREST AND EXPERTISE IN A PARTICULAR 
.CULTURAL OR BROADLY EDUCATIONAL DOMAIN 

C HAVING OUTSTANDING PROFESSIONAL WOVLOCt OF CURRICULUM AN 
PROGRAMMES EITHER IN GENERAL OR IN A PARTICULAR AREA 
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expertise in a particular cultural domain (for example, 
music, fine arts), to an awareness and deep interest in t 
particularly educational domain (such as the education b 
the gifted or the theory of curriculum). It is worth noting 
that the frequency of mention increases as one moves "down" 
the hierarchy. The frequency of mention of each of the 
elements exceeds the number of reports in which the variable 
is mentioned. It is clear, therefore, that more than brie of 
the elements can be mentioned for any given principal. Ah 
analys is of these mul t iple references saggests that a 
relet loriship exists between the mention of this variable and 
the rating given to the principal for whom it is mentioned. 
Farther discussion of this analysis Is given below in the 
section on the llf ferences between differently rated 
principals. 

The description of the p rincipal . As brie might expect, 
the reports contain a fair number of descriptors-- 
adjectives and adjectival phrases about the principals. In 
the sample of 77 reports were 238 adjectives or adjectival 
phrases directly describing; the subject of the report. For 
purposes of the analysis each of these words or phrases was 
written on a separate card, together with a note of the 
number of times it was used, the classification of these 
cards proved a difficult and t ime-cbrisuming task which was 
finally accomplished only by thinking beyond the adjective 
to the characteristic whlrh it described. This yielded a 
total of twenty-eight different characteristics which, in 
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turn, could fairly easily be grouped in five dimensions. 



Table 7 summarizes the results of the classification. 
Its last column shows examples of the descriptors used (In 
this case the ones most frequently used for each 
characteristic). The columns Immediately preceding these 
examples show the number of different descriptors and the 
total number of uses of all the descriptors for each 
characteristic. For example, the adjectives "warm" and 
•friendly" are the two most frequently used of eighteen 
different words or phrases describing the characteristic of 
attractiveness. Several of these words or phrases are used 
more than once in the reports and the total number of times 
one or other of them is used Is 61. The left hand column of 
the table shows the dimensions into which ail twenty-eight 
characteristics can be grouped. 

The detailed discussion of the data in Table 7 IS of 
less interest than three "measures" which can be derived 
from it. These three measures enable one to examine (aj the 
complexity of each dimension, (b) the salience of each 
dimension as a focus for those who were evaluating thi 
subjects of the reports; and (cj the degree of similarity 
among evaluators in choosing what qualities to describe. 
Table 8 shows the results of constructing these measures. 
Referring only to the dimensions described, the top half of 
Table 8 summarizes columns 3 and 4 of Table 7 and adds a 
column showing the average number of uses of each descriptor 
(the total number of uses divided by the number of 



Table 1 



ANALYSIS OF ADJECTIVES OR ADJECTIVAL PHRASES USED 



DIMENSION 
DESCRIBED 



CHARACTERISTIC 
DESCRIBED 



NO.OF 

DESCRt 

IPTORS 



N0*OP 
OSES OP 
ALL 
DESCR- 
IPTORS 



HOST FREQUENTLY 
USED DESCRIPTOR IS) 



ATTRACTIVENESS 16 

SOUND SELF CONCEPT ( 

PERSONALITY GENERAL IMPRESSION HADE 13 
AND 

MANNER FORMALITY 4 
STRAIGHTFORWARDNESS . 5 

ARTICULATENESS 2 

LEVELHEADEDNESS U 

UNOBTRUSIVENESS 6 
ASSERIIVENESS 12 



61 HARM, FRIENDLY 

24 CONFIDENT 

32 NEATLY-GROOMED, HAS 
GOOD SPEAKING VOICE 

4 DIGNIFIED, FORMAL 
IS CANDID, DIRECT 

5 ARTICULATE 
17 CALM 

30 QUIET, LOW KEY 

30 FORTHRIGHT 



VALUES 



ETHICAL BEHAVIOUR 



9 



(0 FAIR 



ABILITIES 



CREATIVITY U 
REASONING i JUDGEMENT 19 
CAPABILITY AT WORK 9 



27 
48 
49 



CREATIVE, 

CONSTRUCTIVE 
HAS GOOD JUDGEMENT, 

INTELLIGENT 
CAPABLE, EFFICIENT 





INDUSTRY 


18 


155 


« 

DEDICATED, 










HARDWORKING 




COURAGE 


4 


10 


COURAGEOUS 




MATURE PROFESSIONALISM 


11 


29 


PROFESSIONAL, WISE, 


APPROACH 








RESPONSIBLE 


TO WORK 


ATTENTION-TO- DETAIL 


4 


4 


METICULOUS 




SENSE OF REALITY 


I 


12 


PRACTICAL, 
HAS COMMON SENSE 




INVOLVEMENT OUT OF 










PERSONAL INTEREST 


J 


23 


INTERESTED, INVOLVED 




SCHOLARLINESS 


1 


1 


SCHOLARLY 




VIGOUR OF APPROACH 


9 


18 


DECISIVE, _ 
BUSINESS-LIKE 



RELATIONS 

WITH 

OTHERS 



ENCOURAGEMENT 4 SUPPORT 4 

CONSIDERATENESS 30 

HILLINGNESS.-TO_ROSPT 5 

GIVING DEFINITE DIRECTION 7 

FACILITATION 1 

VISIBILITY 1 

GENERATION OF TRUST 5 



22 SUPPORTIVE 

151 COOPERAT1VE,PATIENT, 

UNDERSTANDING 

35 • OPEN, FLEXIBLE 

43 POSITIVE, FIRM 

1 IS A FACILITATOR 

19 VISIBLE 

19 CONSISTENT, RELIABLE 
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descriptor! In each dimension). 



The complexity of each dimension may be Inferred from 
the number of descriptors used for the characteristics In 
it-the more descriptors, the more complex the dimension. 
Irrespective of the complexity of the dimensions, thi extent 
to which each is referred to in the reports (and hence Its 
salience for the evaluatora as something to be noted) may be 
gauged by the total number of times It is mentioned. This 
statistic is given by the total number of uses of all 
descriptors. Finally, if we assume that when two or more 
ivaluatora use a particular word, they mean the same thing 
by It, then a calculation of the number of times each 
descriptor is used may be taken as a rough measure of the 
degree to which the different evaluators make similar 
judgements about what qualities in principals are to be 
noted. 

The results of applying these measures are shown in the 
lower half of Table 8. The most complex dimensions appear to 
be "Personality and Manner" and "Approach t? Work", followed 
fairly closely by "Relations with Others"--a finding 
consistent with the view that there is no one best type of 
administrator or style of administration. Less complex la 
•Abilitles"--the evaluatora seem to have a sense of what 
kind of abilities (creativity, reaaonlng and judgement and 
capableness) are worth noting In a principal. Least complex 
of all }» the writer! of these reports is the dimension of 
»" in which ethical ' behaviour Is the only 
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Table 8 



ANALYSIS OF DIMEN5I0N8 DESCRIBED BY THE ADJECTIVES OR 
ADJECTIVAL PHRASES USED IN THE REPORTS 



DIMENSION 
DESCRIBED 



NO. QF r HO.OP-DSESR AVERAGE R 
DESCR- A ALL DESCR- A NO. USES A 
IPTORSN IPTORS N EACH N 



X DESCR- X 



PERSONALITY ( MANNER 


77 


i 


219 


3 


2.84 


5 


VALUES 


J 


5 


60 


5 


6,67 


1 


ABILITIES 


39 


4 


124 


4 


3.18 


4 


APPROACH TO WDRX 


60 


i 


252 


2 


4.2 


3 


RELATIONS KITH OTHERS 


S3 


3 


290 


I 


5.48 


2 



ro^C^Mr^hTMPM^L 1 ^ 8 INTEWRETATI01, S OP THE DEGREE 
TO WHICH EACH DIMENSION IS REPERRED TO 



COMPLEXITY, OP 
EACH DIMENSION 

(NOiOr DESCR* 
IPTORS) 



IMPORTANCE OP 
EACH DIMENSION AS 
A CRITERION 

(no. or uses or 

ALL 

DESCRIPTORS) 



DEGREE OP 
SIMILARITY IN 
JUDGING WHAT 
QUALITIES ARE 
TO BE NOTED 

(AVERAGE NO. OP 
USES Or EACH 
DESCRIPTOR) 



PERSONALITY 
t MANNER 



RELATIONS 
WITH OTHERS. 



2 
3 

4 

5 



APPROACH TO TORE APPROACH TO MORX 



RELATIONS— 
WITH OTHERS 



ABILITIES 
VALUES 



PERSONALITY 
t MANNER 

ABILITIES 
VALUES 



VALUES | 

XEL'NS WITH OTHERS 
APPROACH TO BORX 



ABILITIES 

PERSONALITY I MANNER 
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characteristic described, and that by only nine different 
descriptors. 

"Values" 1b not only the least complex dimension 
emerging from these reports, It is also the least salient. 
What is clearly most salient for these ^valuators as 
something to be judged is "Relations with Others", with 
"Approach to Work" as a close second. "Personality and 
Manner" seems reasonably important, bat "Abilities" are 
referred to far less frequently. 

The greatest degree of similarity among evaluates in 
judging what qualities are to be noted Is In the area of 
"Values", In which each descriptor Is used more than six 
times. "Relations with Others" also scores high on this 
measure, largely because of the almost ubiquitous use of 
worci describing considerateness, A similarly widespread use 
of words describing industriousness results in the third 
place ranking of "Approach to Work" in this analysis, 
whereas the relatively low score of "Abilities" and 
"Personality and Manner" reflect the variety of terms used 
for them. 

Finally, it is worth noting that the least salient 
dimensions ("Values", "Abilities" and "Personality and 
Manner") refer to the person of the principal, and the most 
salient ("Approach to Work" and "Relations with Others") 
refer to the principal at work. If these descriptions tell 
us in some sense how principals do what they do, a different 
kind of data tells us what it is that they do. 
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The Actions of the Principals 

Two hundred arid ninety actions by principals were 
described in the sample reports. The examination and 
classification of this large number and variety of actions 
led to the construction of a four-dlmenslonai framework for 
the analysis. Since the framework Is important for the 
presentation and Interpretation of the results, its 
construction Is decrlbed in detail in the following 
paragraphs. The results of the analysis are presented under 
a second sub-heading. 

The classification of reported actions . A first sort 
of the 290 actions described in the reports yielded twenty- 
one categories, but this result was not entirely 
satisfactory-there seemed to be too many categories and 
there was a sense that actions were being intuitively 
assigned to categories which could not be easily defined. A 
second attempt gave thirteen categories but did nothing to 

r 

dispel the impression that there was a fuzziness in exactly 
what aspects of each action were being given priority as it 
was assigned to a category; 

Accordingly, the task was approached In a different way 
by trying to be specific about which aspects of each action 
were being revealed by the words in the description. This 
led to the clear Identification of four elements In most 
actions. The first resulted from the strong Impression given 
by the reports that principals were often commended for 
acting in consultation with others; For this reason, the 
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first element considered was Who Acts? For this element i 
two : part classification was used; The principal acts alone 
or the principal acts with someone else* A second 
observation about the reported actions was that different 
people were chiefly affected by them* Thus a second element 
became the answer to the question, %[ho Is chi e fly 
af fected?" A three-part classification was used here: the 
action chiefly affects the principal him or herself, the 
action chiefly affects specified others (e.g. parents, 
department heads, a particular commit tee I, the action 
chiefly affects the staff or the school In general, Third, 
the actions were directed at different parts of the school 
operation, thus the third element was discerned by asking 
!t!Si area |s the action targeted? ' . For the 
classification of actions on this dimension the six areas of 
school operation already used In the analysis of the 
features of the school were used (c;f . Page 10 above). 
Finally, the actions differed in kind; some were actions 
designed to provide a sense of direction, for example, while 
others were the kind of actions needed to cope with day to 
day occurrences. In order to distinguish different kinds of 
action, a four-part classification of organizational actions 
was . taken from Handy (1976:198); This classification 
distinguishes between four kinds of action: steady state, 
Innovative or developmental, breakdown or crisis (expanded 
in this analysis to include actions solving particular one- 
time only problems of a serious nature) and actions making 
policy or establishing direction. 



Figure I The Framework for the Analysis of 
Reported Actions 
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Figure 1 shows the matrix constructed to serve as the 
framework for analysis, The combination of all four elements 
In the framework yields 4 x 38 celh. To draw these In a 
different way, as in Figure 2, permits one easily to gi7e a 
unique designation to each cell, Thus "D12" designates 
actions which are of an Innovative or developmental kind, 
which are performed by the principal alone, targeted at the 
area of staff and chiefly affect the staff the school in 
general. Such an action might be "Develo, J a hew system of 
teacher supervision", 

Following the construct Ion of this framework, each 
reported action was re-examined to see If (t could be 
classified in this way, Table 9 shows the results of this 
re-exasil nation, two hundred and seventy of the reported 
actions were fully classifiable by the four dimensions of 
the framework; Sixteen ^uld be partly classified and four 
could not be classified at all, What made these twenty 
actions not amenable to full classification was a lack of 
information about the action In the way It was reported. The 
distribution of the 270 classifiable actions for each of the 
four elements of the framework is shown in the lower part of 
Table 9. Given the apparent utility of the framework in 
distinguishing between different actions, it was used for 
their further analysis. 



Figure 2. Grid Framework for Analysis of 
Reported Actions 
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Table 9 



REPORTED ACTIONS OF PRINCIPALS 



ACTIONS REPORTED AND CLASSIFIABLE 27Q 
ON FOUR DIMENSIONS 

ACTIONS REPORTED AND PARTLY CLASSIFIABLE 16 

ACTIONS REPORTED IN TOO LITTLE DETAIL TO * 
PERMIT CLASSIFICATION 



DISTRIBUTION IN EACH CATEGORY OF CLASSIFIABLE REPORTED 
ACTIONS 



A CTORS 

PRINCIPAL ALONE 245 (9°«7*> 

PRINCIPAL WITH OTHERS 25 ( 9.3%) 



AREA TO WHICH ACTION TARGETED 

PHYSICAL PLANT 11 ( 4.1%) 

STRUCTURES, ROUTINES £ PROCEDURES 51 (18.9%) 

PROGRAMME & ITS ORGANIZATION 84 (31.1%) 

STAFF 74 (27.4%) 

COMMUNITY 40 (14.8%) 

DISTRICT 10 ( 3.71) 



THOSE CHIEFLY AFFECTED 

PRINCIPAL HIM OR HERSELF 30 (11.1%) 

SPECIFIED OTHERS 89 (33.0%) 

STAFF OR SCHOOL IN GENERAL 1.51 (55.9%) 



KIN D OF ACTION 

STEADY STATE 55 {35.2%) 

INNOVATIVE/DEVELOPMENTAL 126 (46.7%) 

CRISIS/PROBLEM SOLUTION 11 ( 4.1%) 

POLICY/DIRECTION SETTING 38 (14.0%) 
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Figure 3; Distribution of Reported Actions 



The reported actions, Sri overall view of what Mrid of 
actions are described in the sample reports Is displayed 
graphical ly in Figure 3; One diagonal line In the figure 
represents one reported action, Thus, the darkest cells in 
the figure are the ones In which the greatest number of 
actions He; Several observations may be made about this 
overall display. It is clear that very few reported actions 
were taken by the principal acting with others, but that 
wore than half of those that were fell intr the category of 
policy or direction establishing actions, It seems, however, 
that not all policy Is established by principals acting with 
their colleagues— a greater number of actions of the same 
kind is shown in cell P9 in which the principal acts alone, 
Also noticeable is the relative absence of actions 1n the 
crisis/breakdown/problem solving category and, with one 
exception, in the class of actions of all kinds affecting 
chiefly the principal him or herself. This one exception 
(cell 010) is probably important, containing as it does, 
actions by which the principal sets up his or her system of 
staff supervision. 

The cells which contain t more actions than any others 
are S 12, D6, S9 and D9. Table 10 shows the number of 
reported actions ih each of these cells and shows examples 
of what kinds of action they are. While the table is for the 
most part self explanatory, it is perhaps worth noting that 
the tense of the verb used in describing the action is a key 
feature in distinguishing between steady state and 
innovative or developmental activities. Thus the example 
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shown In Table ID for an S12 activity ('Carries but a 
'regular and thoroughly developed schedule of staff 
supervision") may be contrasted mth the example given 
earlier of a 012 activity ("Developed a new system of 
teacher supervision"), this distinction, as will be seen 
below, becomes interesting in interpreting the results for 
differently rated principals. 



THE_0IEEE8£NC£S_B£IWEEN_REP0RIS OF 
DIFFERENTLY RATED PRINCIPALS 



Because the reports in all cases conclude with a rating 
of the subject, it is possible to examine whether or not 
there are differences in the way differently rated 
principals are evaluated. Three analyses yield interesting 
results. They concern (i) the unsuccessful principal, (2) 
the actions of successful principals and (3) the dimension 
of personal -professional awareness and competence, 



The^Jnsuceessful Principal 

Little reference appears to be made In the school 
effectiveness literature to poor or unsuccessful 
principal a—perhaps because respondents in most research 
projects are reticent about identifying such administrators, 
It is noteworthy also that Leithwood and Montgomery (1982) 
in their comprehensive review of a large number of studies 



Table 10 



MOST FREQUENTLY REPORTED ACTIVITY CATEGORIES 



CATEGORY HO* OF DEFINITION 
REPORTED 
ACTIONS 



EXAMPLE OF ACTIVITY 



m 



D 6 



S 9 



D 9 



23 



Pi ALONE 
AREA t STAFF 
AFFECTING 
ST/5CH IN 
GENERAL 



CARRIES OUT A REGULAR AND 
THOROUGHLY DEVELOPED 
SCHEDULE OF STAFF SUPER- 
VISION 

PROVIDES GUIDANCE FOR IN- 
SERVICE ACTIVITIES OF STAFF 



20 



IS 



15 



P. ALONE 
AREA: STRUCT, 

ROUTINES, PR. 
AFFECTING 

ST/SCH IN 

GENERAL 



DESIGNED AND PRODUCED A 
SCHOOL MANUAL 



DEVELOPED GOOD NEW ADMINISTR- 
ATIVE PROCEDURES 

INTRODUCED NEW ATTENDANCE 
PROCEDURES 



P. ALONE 
AREAt PROG t 

ORGANIZATION 
AFFECTING 

ST/SCH IN 

GENERAL 



COORDINATES SUPPORT SERVICES 

CAREFULLY MONITORS GRADING 
t EVALUATION PROCEDURES 

ENCOURAGES STAFF INVOLVEMENT 
IN CULTURAL ACTIVITIES IN 
THE SCHOOL 



P. ALONE 
ABEAl -P80G_i„ 

ORGANIZATION 
AFFECTING 

ST/SCH IN 

GENERAL 



INTRODUCED DEN EXTRA-CURRIC- 
ULAR ACTIVITIES 

ESTABLISHED POLICY ( PROCED- 
URES FOR PUPIL ASSIGNMENT 

ASKED THE STAFF TO EXAMINE 
PROGRAMMES FOR CHILDREN IN 
DIFFICULTY 
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are able to distinguish only between "effective" principals 
^ and "typical" or "modestly effective" principals* In the 
present data five subjects were rated as "6" or "7" ("fair" 
and "poor" respectively). Four of these five were school 
principals. 

The reports on these unsuccessful principals do not 
differ markedly from other reports in the attention they 
give to the subjects' schools, but they do differ In two 
other reoects: they are longer (see Table 11 below) and 
rather than describing the actions of the principals they 
describe the absence of actions. Subject 16, for example, 
has a report which notes that he or she "did not know . . , 
Is Ignorant of . . , makes little use of . , , Is not able 
to show ; . ; falls to ; ; , acts without ; ; ;" Subject 37 
"is unaware of . . , neglects ; ; , makes little attempt to. 
.? Subject 69 "is not successful In . , , cannot persuade 
the staff to. , , has not . . , needs to . . , has not 
provided . . " And subject 120 "must learn to . . , heeds to 
, , , has not involved himself in ; ; ;" The words used in 
the report on the only unsuccessful central office 
administrator are similar* This person "has not. . , does 
not , , , heeds to . . , appears hot willing to . . , Is hot 
Involved In . .". Where actions are reprted for these 
subjects, they serve generally to reinforce the dominant 
impression of lack of action: the subject "acts without , . 
" or "makes Inappropriate delegations of . . " Another 
subject spends long hours in the school, but tWi serves as 
an indication not of his Industry but of the apparent fact 



that he has failed to organize his priorities. 

Although the number of principals rated unsuccessful Is 
small, the tone of the reports about them Is clearly 
different from that found in the reports on their more 
successful colleagues. On the basis of what is in those 
reports, It seems likely that a case can be made for the 
assertion that principals who fall do so by omission rather 
than by commission. 

Th e Actions of Successful and 
Very Successful Principals 

The analysis of the actions reported by the evaluators 
was extended so as to Investigate whether different sorts of 
actions were reported in the case of very successful 
principals and reasonably successful principals. The reports 
of those principals who were rated "1" or "2" ("Excellent" 
or "Very Good") were examined separately from those of 
principals rated "3", "4" and "5V The decision to split 
the sample in this way resulted partly from the fact that 
there were only five principals rated in the "Excellent* 
category-too few to permit reasonably valid comparison. Thi 
legitimacy of the decision, however, received striking 
confirmation from an unexpected finding concerning the 
length of the reports and the frequency with which act tons 
were mentioned in them. 

Th e fr e qu e ncy of mention o f-acttonsr For the purpose of 
making some kinds of between-group comparison It was 
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important to know whether a" high frequency of mention was 
the result s triply of the length of reports; Accordingly, an 
analysis was made of the average length of reports in each 
of the seven rating categories and the average number of 
actions reported in them. 



table 11 shows the results of this analysis, while the 
average length of reports shows a curvilinear distribution 
over the rating categories, the number of reported actions 
per report (or per report-page) declines as one moves from 
rating category "I" to rating category "5". What Is 
particularly noticeable Is the marked drop In the frequency 
of actions being mentioned between rating categories '2" and 
"3"Hh contrast to the average page length which remains 
almost the same In these two categories. This finding seemed 
to provide good confirmation of the legitimacy of splitting 
the sample between categories "2" and "3". 

For the purposes of comparative analysis between 
groups, these findings also suggest that the frequency with 
which actions are mentioned needs to be Interpreted 
carefully. The data do, however, raise the Interesting 
question of why evaluatbrs appear to report progressively 
fewer actions as they report progressively lower ratings. 



Table 11 

REPORTED ACTIONS OF PRINCIPALS 

^fl SLF 0 *™ 0 mm m LENGTH OF REPORTS FOR EACH 
OF THE SEVEN RATING CATEGORIES 



RATING N0.0F AVERAGE NO. OF AVERAGE NUMBER OF 

BIN - LENGTH CLASSIFIABLE REPORTED CLASSIFIABLE 

IPALS OF REPORTED ACTIONS 
REPORTS ACTIONS 

I PER REPORT PER PAGE 



2 
3 
4 
5 
6 
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5.4 


27 


5.4 


1.0 


20 


5.4 


101 


5.1 


0.9 


26 


5.3 


101 


3.6 


0.7 
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3.4 


17 


2.1 


0.6 


12 


4.6 


24 


2.0 


0.4 


3 


6.0 
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10.0 









fiteii (a) .Paga.iMgthi_ ca iculit«a by uiihg at a Mauri 
one page ■ approx 300 words 
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Jlonsj. It will/ be recalled that Table 10 
showed the most frequently mentioned types of activity 
across all reports of principals rated "satisfactory" or 
better, fable 12 shows the results of an identical analysis 
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performed separately for each of the two groups of 
principals, it is clear that the most frequently mentioned 
activities in one group are quite different from those in 
the other. The" principals rated T and "2" have reports 
which refer most frequently to steady state kinds of 
activity, while those rated "3«, -4- and "5- bring mention 
of innovative or developmental activities. Moreover, the 
most frequently mentioned kinds of activity for the top 
rated principals concern teacher supervision and staff 
guidance, activities which are In second and third place for 
the second group of principals. Activities having to do with 
structures, routines and procedures are the most frequently 
mentioned for the second group and, although such activities 
are frequently mentioned for the top group, they are In "this 
latter case activities directed at specified others rather 
than at the staff or school in general. These different 
focuses sre reversed in the case of the fourth most frequent 
type of activity m both groups-activities in the area of 
community relations. 

Statistically, the significant differences b( jen the 
kinds of action most frequently reported for each of the two 
groups He predominantly In the kind of action referred to 
(steady state and innovative or developmental) and 
secondarily in the area to which the action is targeted; 
Table 13 shows the distribution of reported actions for each' 
of the groups. The details of the table are perhaps more 
revealing than is the simple designation of the dimensions 
in which the statistically signif icant differences He; The 
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Table 12 ' H0M mm *crm« categories m 
principals rated "i* or *i > and 'p, or-'s 1 

RESPECTIVELY 



RANK 



FOR PRINCIPALS RATED 
•1" OR T 

CATEGORY NO. MENTIONS 




FOR PRINCIPALS RATED 
"3" "4" OR *5* 

CATEGORY NO. MENTIONS 



Table 13 DISTRIBUTI0N op ""omed actions in each of three 

IflOie^U CATEGORIES FOR TWO GROUPS O F PRINCIPALS 
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principals rated T or T evoke no mention of actions in 
the area of physical plant, unlike their less highly rated 
colleagues, but they appear to be noticed much more for what 
they do in the area of district relations. It is also 
noticeable that a larger proportion of the actions reported 
for the second group than for the top group are actions 
which chiefly affect the principal him or herself . It is 
tempting to see these differences *n the differences between 
the principal who has "arrived" and the one who is still 
"trying to make it", From such a perspective, the principals 
rated "3" *4" or "5" are those who have to get their own 
routines and procedures in order, who have to initiate new 
practices in order to improve a less than satisfactory 
situation. Their more successful peers by contrast need only 
to maintain the high quality of their operation. Such an 
explanation is not wholly satisfactory, however, and we 
shall return to the question of how to interpret the 
findings in the last section of the paper. 
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The variable called personal -professional awareness and 
competence emerged because of its distinctiveness and not 
because of the frequency of its mention; the earlier 
discussion of the variable (page 14 abova) noted that it 
seemed to consist of three elements and that it seemed to 
discriminate between principals receiving different ratings, 
Table 14 shows the number of mentions of the variable across 
rating categories, it is to be noted hen that the top two 
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rating categories are the ones In which higher proportions 
of subjects receive a Bit Ion of this variable than" is the 
case in the lower rating categories. The analysis can be 
expanded, however; by considering thr separate elements of 
the variable, These elements were noted in Table 6 (page 14) 
and It was also noted that more than one element could be 
mentioned In one report, Table 15 shows how the mentions of 
each element are distributed across the rating categories 
and also into which rating categories fall those reports 
which mention more than one of the three elements, It Is 
clear from the table that as the mention of elements moves 
from element J to element C (from philosophy, values, sense 
of purpose In life to outstanding professional knowledge In 
a particular area), so the ent s move down the rating 
categories, It is also the case that the reports on 
principals rated T and n 2* are the only ones which include 
mention of more than one of the three elements. 

Perhaps more clearly than any other variable, It is 
this personal •professional awareness and competence which 
seems to distinguish between the excellent or very good 
principals and the rest, Again, however, this conclusion may 
be facile and, like the other findings of the study, needs 
to be discussed in the light of the particular nature of the 
data, 
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DISCUSSION AND CONCLUSIONS 

One way to view the foregoing analyses Is to see them 
as presenting a picture of school principals which seems in 
general to amplify rather than contradict existing findings 
about effective principals and their schools. One rereads 
the reviews of Hemphill it at. (1962), Gross and Harriott 
(1965), Bossert et al. (1982) and Lelthwood and Montgomery 
(1982) in the light of the present findings and one finds 
oneself easily nodding agreement. Hot to attempt a 
systematic* thoughtful, point by point comparison of these 
findings with those of others is to come face to face with 
two questions about comparability. 

The first is the problem posed by the existence of 
different classification schemes. In order to make a 
definitive statement about the congruence between the 
present findings concerning the actions of principals and 
the comparable material in Lelthwood and Montgomery's work, 
one would heed to reexamine each of the 290 actions 
mentioned In the present reports and reclassify then In 
terms of the thirteen sub-dimensions and the twenty*three 
sub-sub dimensions used by those authors. Conversely, If one 
wanted to explore the unusual and perhaps disturbing finding 
in the present study that the best principals are not 
reported to take Innovative actions as frequently as the 
more "modestly effective 1 principals, then the Lelthwood and 
Montgomery data would have to be subjected to analysis by 
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the present framework. 

the second problem is more subtle, less easily 
recognized! arid yet, of major importance. It emerges from 
the realization that in discussing the present findings one 
is constantly tempted to see them as describing principals 
and what they do. In reality, this is not what the date 
describe at all. What they describe is what super iritehdenta 
say about principals and how superintendents Judge 
principals. 

In one sense, of course, this provides exactly what 
Rowan et at. (1983) have called for in suggesting the need 
for grounded definitions of excellence which reflect 
practitioners' understandings of the term. But will the two 
be comparable? is the soil in which the practitioners' 
understandings are grounded the same as the earth from which 
the research -generated findings spring? Before that 
question is dismissed as silly, it is worth considering two 
other questions: (1) For whom were the present reports 
written? Arid (2) Sri superintendents more rational than most 
of us when they make evaluations in the course of their 
dally work? 

The reports are not written for wide consumption. 
Three, possibly four copies will be made of most of them: a 
copy is lodged with the Ministry, a copy is' often given to 
Scool Board chairmen to keep them in touch with whit is 
happening in the schools, the principal who is being 
evaluated receives a copy and he or she nay use a copy in 
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applying for another position (a fact which almost certainly 
means that the writers of the reports wMte with ah e/Bte a 
fellow superintendent who may one day be considering 
principal X for a job). Given this restricted audience it 
is perhaps rather remarkable that the reports are often so 
long and that superintendents obviously put care into their 
composition and are at pains to provide not Just overall 
judgements, but judgements about the quality of almost every 
aspect they discuss. Could it be that these reports are not 
merely reports of evaluations but also a means by which 
superintendents establish and reinforce norms for the 
profession and for the princlpalship? The suggestion Is hot 
that these reports are deliberately composed charters, but 
that they are partly conscious, partly unconscious 
formal Izations of what is to be seen as a good school and a 
good principal, 

if this is the case, and If one function of the reports 
is to establish norms, to formalize and emphasise what it is 
that superintendents believe, then It is reasonable to ask 
how rational is their approach to evaluation; On the surface 
there is an evident rationality of approach in the reports. 
And yet; a nagging question is posed by the finding that 
reports on principals who are rated as "Excellent" and "Very 
Good" contain twice as many accounts of action by the 
principal as t» the reports on their less highly rated 
fellows. 

One possible explanation is that; like most of us who 
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commit ourselves to something we like, superintendents have 
no trouble justifying what they see as a good principal. 
What this explanation means Is that the superintendent knows 
who his excellent principals are and in looking at then and 
their schools is able to find ample evidence to show that 
they are excellent. 

If this interpretation is correct, then the present ' 
findings do not tell an aspiring principal what he should do 
If he wants to be favourably rated, they tell the favourably 
viewed principal what It Is about him or her that will be 
noticed. They do not enable the researcher to "predict that a 
modestly effective principal will become excellent if he or 
she does thus and so, they allow the researcher to predict 
that If a principal la rated 'Excellent" by the 
Super Ihtendent, the superintendent will be well able to 
describe certain things about that principal. The difference 
may be trivial if we want to know what Is associated with 
excellence. It is probably not trivial If we want to develop 
excellence. , 
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